In order to store or transmit digital image information efficiently, it is required that digital image information be compressively coded. In the status quo, as a method for compressively coding the digital image information, there are waveform coding methods such as subband, wavelet, fractal, and so forth, as well as DCT (Discrete Cosine Transform) typical of JPEG (Joint Photographic Coding Experts Group) or MPEG (Moving Picture Experts Group).
Meanwhile, as a method for eliminating redundant image information between adjacent frames or the like, there is a method in which inter-frame prediction is performed using motion compensation by representing values of pixels of a current frame by difference values between these values and values of pixels of a previous frame, and a difference image signal of the difference values is subjected to waveform coding.
An interlacing scanned image signal such as a video signal in a current television forms an interlaced image. In this interlaced image, one frame is composed of odd-numbered and even-numbered fields, the scanning timings of which differ from each other. For this reason, in the interlaced image, whether or not correlation of pixel values between adjacent scanning lines is high depends on largeness of motion of each object displayed on a frame (each object having an arbitrary shape in a frame image).
Accordingly, when coding an image signal corresponding to the interlaced image, in a case where there is high correlation of pixel values between scanning lines, motion compensation and waveform coding are performed to the image signal frame by frame, while in a case where there is low correlation of pixel values between scanning lines, the image signal is divided into odd-numbered and even-numbered fields, the image signals of which are subjected to motion compensation and waveform coding field by field.
In recent years, for reproducing an image signal for each of objects in a frame image while increasing efficiency in compressing an image signal, a coding method has been conceived, wherein image signals of the objects are compressively coded and transmitted separately for each object. A coded image signal which has been coded by this coding method is decoded at the reproducing end in a manner adapted to the coding method. Specifically, in this decoding, coded image signals of respective objects are decoded, and the resulting decoded image signals of respective objects are synthesized, to produce a reproduced image signal. Then, based on the reproduced image signal, a frame image comprising respective objects is displayed.
By using the coding method for performing coding to the image signal for each object as described above, it becomes possible to combine objected freely to generate a composite image, whereby moving images can be reedited with ease. In addition, a moving image comprising only objects with high importance can be displayed without reproducing objects with relatively low importance, depending on a degree of congestion of a communication path, performance of a reproducing apparatus, or preference of a viewer.
For coding an image signal of an object, (an image having an arbitrary shape) a waveform transform which performs signal processing adapting to a shape of the object, for example, shape adaptive DCT, or waveform transform to a padded image signal, is employed. In the method, specifically, padding is performed to an image signal forming an image space (rectangular region) by replacing values of pixels in its insignificant region with padding pixel values obtained by a prescribed method, and then a conventional 8×8 cosine transform is performed to the padded image signal. The insignificant region is a region outside an object in the rectangular region, and comprises pixels which have no values for displaying the object. In other words, an image signal of the insignificant region comprises so-called insignificant sample values. Also, the 8×8 cosine transform is a waveform transform which performs cosine transform to the image signal of the rectangular region for each image space comprising 8×8 pixels.
As a method for eliminating redundant signals between adjacent frames, there is a method for obtaining a difference between an image signal of a target macroblock to be coded and its prediction signal for each image space (macroblock) comprising 16×16 pixels, as a prediction error signal (difference signal). Here, it is assumed that the prediction signal is an image signal of a prediction region obtained by motion compensation. The motion compensation process detects a region comprising 16×16 pixels corresponding to an image signal in a coded or decoded frame where a difference between the image signal of the target macroblock and the image signal is the smallest, as a prediction region.
In some cases, however, the prediction region also includes pixels (hereinafter referred to as insignificant pixels) which have insignificant sample values. In such cases, when the difference between the prediction signal of the prediction region including insignificant values and an image signal of the target region is computed, difference values tend to be very large, since the sample values of the insignificant pixels are not always optimal prediction values for smaller difference.
As a solution to this, padding is performed to the image signal of the prediction region by replacing insignificant sample values by prescribed padding values, and then a difference between the padded prediction signal and the image signal of the target macroblock is computed as a difference signal (prediction error signal), which is subjected to transformation for coding. This padding of the prediction signal can suppress the difference signal.
In the prior art pixel value padding, as a padding value for an insignificant sample value, sample values of significant values (pixels having significant sample values) adjacent to the corresponding insignificant pixel in the vertical and horizontal directions, respectively, are averaged, and the resulting average is used. By the use of the average of the sample values of the significant pixels, high-frequency components into an image signal (prediction signal) in a padded image space can be suppressed, thereby increasing coding efficiency.
However, if padding which uses the average of the significant pixels adjacent to the insignificant pixels in the vertical and horizontal directions as the padding value is applied to the image signal of the interlaced image, high-frequency components of the image signal increase, which will be described below in detail.
In the interlaced image, especially in a case where motion of an object is large, there is low correlation of pixel values between adjacent scanning lines. FIG. 18 schematically shows an example of an arrangement of pixel values which shows the low correlation of pixel values between adjacent scanning lines in the interlaced image, as arrangement of pixel values in an image space 301.
The image space 301 is an image space comprising 8×8 pixels. In the figure, each pixel is represented as a square. Pixels (for example pixels 303 and 304) of hatched squares are pixels (significant pixels) having significant sample values, and pixels (for example pixel 302) of squares which are not hatched are pixels (insignificant pixels) having insignificant pixels. The number of each square indicates a sample value of the corresponding pixel.
Since scanning timings differ from each other between odd-numbered and even-numbered fields in the interlaced image, in a case where motion of an object in an image is large, it looks like the object has a shape of two outlines in a frame composed of these two fields, as it can be observed in the image space 301 in FIG. 18. In this case, for example, the significant pixel 304 is adjacent to its upper and lower insignificant pixels.
Next, a prior art padding method and its padding result will be described with reference to FIGS. 18 and 19.
In the prior art padding method, three steps are performed.
In a first step S1, sample values of insignificant pixels are replaced with sample values of significant pixels adjacent thereto in the image space 301, for insignificant pixels aligned in the horizontal direction, in the order from inward to outward of the image space 301. Then, sample values of insignificant pixels in the image space to which this horizontal replacement process has been performed are replaced with sample values of significant pixels adjacent direction, in the order from inward to outward of the image space. Process in the first step S1 thus performed to the image space 301 results in an image space 350 comprising pixels having sample values shown in FIG. 18.
More specifically, in the horizontal replacement process in the first step S1, sample values of insignificant pixels on pixel rows 354–359 in the image space 350 are replaced with padding values (sample values of significant pixels). In addition, in the vertical replacement process in the first step S1, sample values of insignificant pixels on pixel rows 352 and 353 are replaced with sample values of significant pixels and padded pixels (insignificant pixels whose sample values have been replaced with padding values) on the pixel row 354.
In a second step S2, in the reversed order of the first step S1, sample values of insignificant pixels are replaced with sample values of significant pixels adjacent thereto in the image space 301, for insignificant pixels aligned in the vertical direction, in the order from inward to outward of the image space. Then, sample values of insignificant pixels are replaced with sample values of significant pixels adjacent thereto in the image space to which vertical replacement process has been performed, for insignificant pixels aligned in the horizontal direction, in the order from inward to outward of the image space. Process in the second step S2 thus performed to the image space 301 results in an image space 351 comprising pixels having sample values shown in FIG. 18.
More specifically, in the vertical replacement process in the second step S2, sample values of insignificant pixels on pixel columns 362-365 are replaced with padding values in the image space 351. In addition, in the horizontal replacement process in the second step S2, sample values of insignificant pixels on pixel columns 360 and 361 are replaced with sample values of significant pixels and padded pixels on the pixel column 362, and likewise, sample values of insignificant pixels on pixel columns 367 and 368 are replaced with sample values of significant pixels and padded pixels on the pixel column 365.
In a third step S3, as shown in FIG. 19, sample values of pixels in the image space 350 obtained through the first step S1 and the corresponding pixels in the image space 351 obtained through the second step S2 are averaged, which results in an image space 380 comprising pixels having sample values in FIG. 19.
Resampling which divides pixels forming the image space 380 into pixels corresponding to odd-numbered and even-numbered fields based on an assumption that the image space 380 is a frame, results in an image space 381 comprising plural pixels corresponding to the odd-numbered field and an image space 382 comprising plural pixels corresponding to the even-numbered field asa shown in FIG. 20. In the image spaces 381 and 382 corresponding to these respective fields, sample values of pixels are non-uniform, which introduces high-frequency components into image signals in image spaces 381 and 382.
In addition, this problem associated with the padding occurs in the non-interlacing scanned image (progressive image) as well as in the interlacing scanned image (interlaced image). Specifically, in many cases, the progressive image has a stripe pattern, and for the case of the progressive image having the stripe pattern, the above resampling is performed in such a manner that plural pixels forming a frame (image space) are collected into a stripe portion and the other portion in the image space, to form image spaces corresponding to the stripe pattern portion and the other portion, and then the resulting resampled image spaces are coded.
In this case, if padding is performed to the image signal of the progressive image having the stripe pattern by the prior art padding method, high-frequency components are introduced into an image signal corresponding to the resampled image space, leading to degraded coding efficiency.
Further, a similar problem occurs in padding methods other than the above, which computes padding values using significant sample values adjacent to insignificant pixels in the horizontal or vertical direction.